\section{User Survey}
\label{sec:survey} 

We have developed a testbed user survey to gather user and operator input on how well testbeds meet user research needs and what improvements are needed in the future. The survey was advertised to our local DETER user community and to several mailing lists where networking researchers subscribe (e.g., IEEE COMSOC TCCC list) in March 2011. We have received 122 responses: 88.5\% (108) of respondents said they use a testbed, 27.8\% (34) said they operate a testbed and 8.2\% (10) said they do not use testbeds.  Out of 112 respondents that do use or operate a testbed,  27.6\% (31) use Emulab, 58\% use DETER (65), 33\% (37) use Planetlab, 8\% (9) use in-house testbed and smaller percentages (1--4 respondents) use other testbeds. 

Out of ten respondents that do not use testbeds, seven said they are not familiar with testbeds. There were six responders who said they use a simulation instead of a testbed. Six users noted they believe testbed experiments were difficult to set up. This indicates a need for broader advertising of testbeds to research audience and development of user tutorials to lower the cognitive entry cost. 

We asked those responders that use a testbed to estimate what percentage of time they spend on experiment design, developing a prototype, developing tools to create realistic settings for experimentation, running experiments (without debugging) and debugging experiments. Results are shown in Figure \ref{ustime}. Almost a quarter of user time is spent developing a prototype and around one fifth of the time is spent in experiment design, running or debugging, leaving 15\% of time for tool development. Better testbed support for experiment design, sharing and reuse, tool sharing and experiment monitoring and debugging could cut down up to 55\% of current experimentation time. 

We also asked all responders to rank the importance of several improvements on a scale \{"unimportant", "somewhat important" and "important"). We have received 114 replies, which we summarize here. Monitoring and debug support were ranked as important by 85\% of respondents, as were tools for specific research areas (e.g., traffic generator for IDS testing). Learning materials such as tutorials, packaged experiments and class materials were ranked as very important by 63\% of users, and a way to publicly share experiments was ranked as very important by 62\% of users. Only 12\% of users wanted testbed service to improve, e.g. to offer more nodes or faster experimentation. These answers indicate that testbeds should enrich their experimentation environment with domain-specific tools and lifecycle management.

Finally, we asked users to tell us their preferred approach to evaluate their research solutions. Simulation is preferred by 36.9\% of users, in-house testbed by 26.2\%, network measurement by 19.7\%, network deployment by 19.7\%. Public research testbeds come in the fifth place with 17.2\% respondents, while 12.3\% use theoretical modeling. This paints a difficult competition for public testbeds.  In-house testbeds presumably offer some security or tools that public testbeds do not, and theory and network measurement offer features that are not compatible with testbeds. But simulation and real network deployment are direct competitors with public research testbeds. We believe testbeds are losing the battle here because the advantages they offer over one of these approaches are smaller than the other approach offers. For example, network deployment is difficult and testbed experimentation is somewhat easier, but simulations are far easier to learn, setup and reconfigure than both of these approaches. On the other hand, simulation fidelity is lower than that of testbed experimentation, but real network deployment has far greater fidelity and realism than both of these approaches. Thus users that seek ease, repeatability and control prefer simulations to testbeds, and those that need realistic environment gravitate towards real network deployment. To improve their standing in the future, testbeds need to improve both their usability and their fidelity. 

